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ABSTRACT 

Perhaps no facet of long-term care (LTC) research has 
received a^ much attention as the evaluation pf community-based LTC 
demonstrations. Recent thinking has called into question the validity 
and policy relevance of the findings of these studies, yet their 
efforts will continue to receive strong f iiiancial biaicking. The time 
has come for LTC analysts to rethink the place^^of evaluation research 
as a tool for LTC policy development and to strengthen the methods 
for such development . Following the introductory analyses of LTC 
evaluation research, this paper' describes a quasi-experimental 
research design, cohort anaj|.ysis, used to. evaluate the Hospital 
Discharge Demonstration Pro ject Chddp) funded by the State of 
Illinois, which provided short term^ in-home! assistance to impaired 
persons, sixty. and older, upon discharge from Chicago hospitals. The 
benefits of the quasi-experimental cohort analysis,' when an 
experimental design cannot be used^ are explored; specific topics 
discussed include sample comparability (in background and 
characteristics), sampling techniques, and the effects^of political^ 
and organizational realities on the study design. A fi|||a,l section 
compares the benefits of traditional, comparative cost and 
effectiveness evaluations of community-based aTnd institutional LTC ^ 
and the quasi-experimental cohort analysis design. IMCF) 
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Perhaps no facetof long-term care.(LTC) research has received 
as m'uch attention for as extended a period of time and wLth as 
l^r.ge an Investment of fundfi a§ the evaluat Ion of community-based 
LTC (demonstrations. The first studies 1 n t h 1 s-^^ a r e n a were 
conducted almost twenty years ago (1,2) and the volume of 

research acitlvlty since tb^at time ha^ grown at a steady pace with 

\ ' ' ^ ' ' ' . ■ t ' • 

I ^ • ' I 

the support of federal funds authorized under severaiji 

different pieces of legislation. (9,10) ^ 

A parad.ox becomes evident wh,en the current status of 

research on this topic Is assessed. On the pne hand, recent 

thinking has called into question the validity and policy 

relevance of the findings generated by studies completed to' date. 

(6,7,4,8,9). On the other hand, it appears llkel y that 

res-earchers will not' only continue to conduct evaluations in this 

area, but that their efforts will continue to receive strong 

financial backing. A representative of the -Officevof the 

Assistant Secretary for Planning and Evaluation at the Department 

V . ■ . ' 

of Healthand Human Services has estimated that twenty million 
tederal d o 1 1 a s alone had. he^^n~^^^h^^ as of 19 80 for LTC 

demonstrations covering each o^; the nex-t two fiscal years (FY'83 
and FY'8«4). . . ' . 

Clearly the time has come for LTC analyses to rethink the 
(jl^^e of evalQatlon research as a tool for Itiforinlng the 
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development of LTC policy and to strengthen the methods available 
for doing so; It is toward these goals which the present paper 
is directed. Two specific- questions will be discussed: First, 
what are tUe appropriate issues with which future eval^tions of 
non-institutional LTC should be conce rned ? ; an d , second, can the 

« 

methodological problems which have plaqued studies in the past be * 
resolved sufficiently enough to rende r future research capable of 
generating valid findings to these questions? 

The principal criticism which has been voiced conberning 
existing evaluations is- that they have typically generated 
findlags .,1th Hm.lted utility for planners in LTC. The problem 
has been l o pronged: not only have, the overwhelming majority of 
studies focused on only one very narrow set of policy questions 
-- those related to the comparative cost and effectiveness of 
institutional and no n -1 n s t i t u t i o na 1 care but analysts now 

agree that studies on this to^J^are ot little practical value to 
policy makers. In the f i r s X plic e BTany demonstrations have been 
designed to illustrate the pa-c i t y of community-based LTC to 
substitute for 1 n s t i t u t i o na 1 iV a t i o n , yet the clients actually 
^ served .have been shown to be significantly less impaired than 
persons requiring institutional placement. Additionally, while 
some studies have shown non-institutional LTC to be a less costly 
alternative than institutionalization (others have not), it' does 
not . necessarily follow that savings |Would result from expanding 
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their availability* Analysts warn that latent demand for such 
services among persons who would never u^se^ ^sources now devoted^ 
to institutionalization is high a\ad that, consequently, cost-\ ^ 
containment^ represents a .w eak;. j u s .t i f i c a t i on' for moving in the 
direction of greater empha^sisf on community-based LTC, (5,3,8) - 
And, finally, Maddox and others have contended that there exist a 
host of factors which are Jikely.ta be m\ore significant 
determinants of the distribuVtion of public resjburces in .LTC that! 
the cost ^atid; effectiveness of alter'nati\^ apprqaches to' providing 

service. (7,6) These ^critics cite cult^ural preferences in 

^\ 

support of family responsibility for elders, the need forifreedom 

■J . . ■ - ' • 

of choice, and recognition that n on- i n s t i t u t i ona 1 care shifts 

much of the burWen of costs from, the puJ>li.c to ifhe private sector 

asthreesuchinfluences. ' , 

It has been arguemfents such as these which have led Gurland, 

Bennett and ^Wilder to suggest a redirection of future LTC ev^lua 

t.ion erforts. (6) In their article fr'pm the Journal of Social 

Issu ers , these authors contend that ''the most challenging issue i s^ 

not whether n on- in s t i t u t i o na 1 alternatives are as effective as ' 

Institutional care, but rather whether it is feasibl^e, within the 

resources available, to provide everly client with a full range 

of optlorib for care together with good advice as to what the 

various options can offer to client and family (in terms of 

services, quality of life ^nd outcomes.)" (p. 67) In sum, 
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although ■ studies to date have not generated information of 
significant value for LTC policy-makers, there exist a wide range 
.of issues which c^n be addressed through evaluative research- and 
which are critical to the planning of effective LTC j)olicy. 

We now ct)me to the second stage in our analysis/ As you 
will reca^ll, p o 1 i cy- r e 1 e v a nc c was not the sole ground on which 
critics have- found fault wi th \existing evaluations. Indeed, fo 
each point raised contesting the utility o^f findings which hav 
been presented, those who have reviewed the evaluation litetature 
in^ this field could probably list at least one equally 
tLoublesome aspect with the methodologies which have been 
employed. Overall, these^design centered criticisms fall into 
two categories; the. non-experimental nature of the research 
mcichods employed and the weakness.and non-standardization across 
sLudie& of the measurement instruments, used for data collection. 
As Dl . Caro has addressed the q\iestioti of measurement, I will 
fo<.u6 Lhtt remainder of my presentation on a discussion of design 
issueci. Specifically, I would like to offer a description of the 
quasi -experimental de&ign I employed in the evaluation^of a LTC 
demonstration recently completed in Chicago -- c^ne w'hich I feel 
has the potential to provide valid findings to many of the 
research questions which should be^the central concexn of 
evaluators as the next phase of research in thisj arena unfolds. 
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The particular program to which I will be referring is t]tie 
Ho6pital Discharge Demonstration Project, or HDDP. Funded by the 
State of Illinois, the HDDP p.x^ovi&ed short-term, in-home 
assistance to Impaired persons sixty and older upon discharge 
from nine participating hospita^ls on Chicago's southeast side; 
The University of Chicago Hospital served as the lead facility 
through which the operational and research elements of the 

project were adminstered, - 

Given the benefits hypothesized to result^ from the services 
provided, ethical considerations prevented, as they have in most 
such studies, the implementation of an experimental research 
design with random assignment of applicants to" treatment and 
control ^conditions. As you are no doubt a ware random assignment 
provldeb the .researcher with the assurance^ that post-treatment 
d 1 t 1: e r e u c e^s between research g r o u p s a re. he r e s u 1 1 of services 
received by a project's clients and not of pre-existing 
d 1 t f c: i. ci.c e^s betiween the groups on characteristics related to. the 
ouLcoine m e a o a t e s . Wh^n it is not possible to use an experimental 
approacii, the researcher must select a quasi-experimental design 
whlcti tioL only peimlLs control for such p r e --t r ea t men t differences 
but vvtiici. a 1 o o allows ^coptrol fcri: as many other factors as 
pocsall. It. »whi., h la jl ght. confound the researcher's ability to 
aLLtibucc ouLcomcs to the treatment being studied. i ■ ' 



ERIC 



-6- 



The benefit of the particular quasi-experimental design 
selected for evaluation of the HDDP, cohort analysis, lies In its 
capacity to at least produce a control group which is highly 
equivalent to treatment recipients on critical control variables. 

As Cook and Campbell describe it in their book ^uasl- 

" ' -'it 
Experimentation , a cohort design involves the comparison of 

groups of respondents who follow on'e another through ^ formal or 

informal institutio'n in which a treatment innovation is 

- \ 

introduced. (A) In thisway, a cohort which enters the 

institution during the period in which the services are in place 

can be compared with another which preceded or followed itbut 

for whop no such ser\^ice was available^ The value of the 

technique lies in the' fact that it is often reasonable to assume 

that a particular cohort di^ffers in only^minor ways from its 

^ ... ' 

contlguoub cohorts. The shorter the time span betwfeen the 

selected cohorts, the more reasonable this assumption becomes. 

^ The cohorts studied in evaluation of the HDDP were patients 

whc. were admitted to the participating hospitals during the 

winter and spring months of 1980 and 19 81. Outcome^^on a sample 

of 190 riDi> F^c 1 1 diit s referred for service between March and May of^^ 

J981 were compared with outcomes evidenced in the medical records 

of 105 patients who had been discharged between March ^nd May of 

1980, the winter before the project's inception, from one of four 

randomly selected participating hospitals. 
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Data on the achieved comparability of the two study samples 
supports the recommendation that cohort analysis represents a 
quasi-experimental approach- with significant potential to produce 
valid findings concerning the Impacts, of n o n-1 n s t 1 t u t 1 ona 1 
alternatives In LTC. These data Indicate that the two HDDP study 
groups were highly comparable on several Important background 
characteristics. To begin with, the samples were not 
statistically distinguishable^ In racial or age' composition 
although the control cohort did appear to have a significantly 
higher proportion of males. While t h e a g e- d 1 s t r 1 bu 1 1 o n of the 
controls appeared skewed toward the younger categories, a T-test 
of the difference In jpean age between groups Indicated a lack o# 
statistical significance. | 

The data also revealed no differences In the medical 
haracterlstlcs| of the two groups. Net only did they tend to 
ffer a similar a ve r a g e n um ^ e r of c hf on i c d is a b 1 1 n g m e d 1 c a 1 
conditions, they were also remarkably similar with respect to the 
nature of the illnesses for which they had been admitted. Almost 

'^i e ' • " ■ ■ ■ , • ' " 

twenty-flye percent .of patients in both groups were admitted for 
treatment of some f o r m o-f c a n c e r while, an additional ten to 
fifteen percent were admitted for conditions Involving the 
cdrdlovciscular -System. The two samples were also equivalent with 
respe-ct to the frequency with which they suffered from medical 
conditions other than that for which they had been admitted. 
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The one characteristic cm which a direct test of equivalence 
was not possible but would have added greatljr to the strength of 
the analysis concerns the 'level of formal and informal supports 
available to the two groups. The inability to pursue an analysis 
of the comparability of the/two groups on thie variable stemmed 
from the fact that data for the controls was limited to the 
information which was contained in their medical record. 
Clearly, if the cot^trols had been receiving some amount of 
assistance from formal or informal source^ at levels equal to or 
greater than the quantity of service provided to treatment group 
'members through the HDDP, the interpretation of the findings on 
the outcome measures would have been significantly confounded. 
However, the equivalence of the two grdups on characteristics 
associated with the need for service for which data were 
available lends support to the assumption of cohort analysis that 
the two gioups were probably similar with respect to this and 
other vaclables for which no information was obtainable. 

AC this point it *ls instructive to point out that the 

( 

potential of cohort analysis to generate equ\^valent study samples 
la mediated, as it is in the case of any q u a s i -e xpe r i m en t a 1 
situatioQ, by external political and /Organizational realities and 
by the ingenuity of the researcher in. .taking advantage of 
whatever idiosyncratic featur^es of the situation will help 
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maxl^mlze the assurance of comparability. In the case of the 
HDDP, for example, external factors worked to the researcher's 
advantage in c o m b a 1 1 i n g t he ,d i f f i c u 1 ty , just mentioned, of not 
^eing able to directly test equivalence on t H^e level of 
alternative sources of assistance available to both groups. 
Specifically, there was strong reason to presume the absence of 
any formal community -based services of which member's of the 
control coh-ort would have made use had they b e e n a va i la b 1 e . 
First, intake into the state's Community Care Program was 
restricted at the time. in which the control group was, sampled. 
Furthermore, the level. of free social services generally 
available to the relatively impoverished residents of the study 
area is recognized to be very low. Thus^ faced with no "access to 
the. state's primary soifrce of fion-income tested community-biased 
LTC, it was likely that members of the control cohort had no 
^Lhcr formal sources of aid to which they might turn. 

ia addition, it was possible for the researcher to maximize 
cue equivalence of the study groups by i s t i tf u t i ng ' a purposive 
sampling technique for selecting the medical records w.hich would 
provide the c o n l rol g roup data. Specifically, only patients who 
had been reterted for discharge planning to the social service 
departments of -the hospitals participating In the sampling 
process were included in the control condition. It was reasoned 



that elderly persotis in need, (yf 'dlsVharge plannlngv woul^d not f^\y^ 

be likely to r,equlre some po s trd 1 s charge LTC assistance, but that 

they iwo^uld also be likely to have few formal or Informal sources 

* ■ . ' ' ' . "• * 

of aid through which to meet t h e s e n^e e d s . . Indeed, it was these 

• t ) ' . y : ■ 

two criteria which guided the referral of pa^t.lent s t o the HDDP 



itself, and thus to thrg" treatment group of the study. 

The point being made here. Is that while' cohort" analysis has 
^ bui 1 t -1 n capacity for generating a control group with 
substantial similarity to a: d e mo n s t ra 1 1 o n 's g 1 1 e n t population, 
^"^e researcher may be able to build'on this base, in such a way as 

to further, strengtheii the a-ssuran^ce of comparability, " 

. ■ ' ^ ■ . •■ ' • ■■ ' .\ ^ ' \ ^- , " 

It i^s, of course, true that the equivalence of treatment and 

control groups is only one criterion oh which, the validity, of 

^^'actributlng outcomes to services pro ^HL-d'^^^^«ni d-er a' quasi- 

experimental approach to evaluation i\s ba,se,dT]Wd threat 

is represented by^ the fact that the study groups in a cohort 

design are sampled from two d 1 s 1 1 n c t p e r 1 o d s 1 n time an-d-€4hey 

may, as a result, have be^n^dlfferentlally stib.jecjte'd to- 

'contextual influences related tjo the study's outcome variables, ^ 

^thereby . c 8 n f o u n d 1 n g*' interpretation of the r e s li 1 1 . This 

. dif f Icul t y 1 s known as .an histaxy effect in the jargon of the 

field, and Cook and Campbell discuss several me thodological ways 

^•around it 1 n t h e 1 r b o ok c 1 t-e d e a r Tl e r . ; 'Wh a t i $ m o r e , ' - 1 h e 
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; re'8.f arche r 1 s o f t e.ri "a position to identify ahead- 6-f time 'the 

.• : -Xk'. ' ■ - - ' ■ _ • • 

possibility of <i'onf6hnS'^ f rjba • the" hliStorical context of a study 
and thus to-* determine thf wisdom of apply Ingr cohor t : ana lys i s 
^ under , existing. coridi;tioas, , , 

;^ ;>And now we have . feome 'full-* ci rc;! e. For, while a cohort 

* * . . * '■ - 

, ■ ' ' "" ■ , ' ' • . , ■ ',1 . . ■ • 

, 'a.pproach to quas i--exj>erimentat idn ^e<;.eives a . positive rating on 

Its methodological s oiiiiidne ss , w e must^as^k whether or not it is 

; V ^ t / ' _ ^ , , . , . ■ . . ' • / ^ ■ ' ^ , ■ 

J capable ofprdyidlng d^t'a on;\pollcy questions of more Jnterest to 
LTC analysts than t;hose t'el^ to c o m p a r a t i'\re cost and 
effectiveness o f communi t y-bisied 'alad ins t i t u t iona 1 LTC?/ Here the 
otherwise confident, appraisal of the success of this [design as 
applied in t^e context of the.JlDDP becomes slightly^i^shed, in 
that the range of pr o j e c t o u t c o m e s which c o u 1 d b e .s t ild i ed wa s 
limited. Th'is si'tuation resulted "from a fact already cited, that 
data for the control cohort was restricted to t ha t ^ whi ch was 
available in their medical records. Concern for patient 
confidentiality prevented release of the names of the individuals 
whose medical files were used for generating. control.data .and, 
consequently it was not possible t-o contact them for the purpose 
of expanding the outcomes studie*d. 



The lesson to be drawn here is that the potential Of jobhort 
analysis to address po li c yr^r e le van t questions depend/s -o^n,' ,th e 
capacity of the researcher to maximize the^ data available ^on jt-he 
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control sample. One way of achieving this end iij future studies 
would-be to select f or t he c o n t r o 1 condition, the.co^o^rt whose 
entrance into the institution from which the study will be 
conducted post-dates termina t ion ^ of/ the demonstration services. 
In this way, permission for participation in the study can be 
obtained directly from the patient s ' themselves an<i the;range of 
information collected on th^m. greatly expanded. A similar end 
could be achieved in cases where there"* is no identified point of 
termination of the? demonstration- services by beginning the 
research early enough to employ as controls individuals who enter 
the stu<iy cite prior to the initiation of demonstration services. 
Barring either of thesealternatives the researcher must be as 
compelling a s. pos s ible in his or her^equest for access to the 
names^and address of persons whose administrative records are 
chosen to represent th^' control condition. / J ' / 

The utility of. cohort analysis in LTC evaluations receives 
f.urther 8upport> when it is recognized that many no n-inS turi tional 
LTC programs will be' provided through hospitals the setting 
most a 5^ e n a b 1 e t o a p ip 1 Ic a t i o n o f ; t h e d e s i g n . • -;^S t af s sen a n_d 
Holohan's 19 81 review of past demonstrations reveals that at 
^least ten of fourteen 'small sca^le p r o j fe c t s. \ d e s c r ibe d were 
hospital basest. .this makes sense given the greater vulnerability 
of the recently discha'rgjed elderly patient and suggests, that the 
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hospital mdy be a setting frequently s e 1^ c t ^<f , f o r studies 

*( 

' conducted •in years to come. 

Finally, it must be noted that-valid and policy relevant 
findings from individual demonstrations serve^ little purpose if 
the results can not be synthe^sized with those of other projects 
to provide a more In-depth and comprehensive understanding of the 

^different outcomes associated with> a^er-na tive approaches^ to 
providing com mun i t y-ba s e d " LTC. Thus, process evaluations^ 
clarifying the specific nature of alternative strategj.es for 
provision of- non-institutional service are sorely needed as are 
the development of valid and reliable measurement instrumentss and 
their standardized application across studies. At the same time, 
such comparative analyse scan not even beattempteduntil the 
findings generated through individual projects begin to approach 
an acceptable level of validity and utility. In this respect, 
more frequent application of cohort analysis, wherever possible, 
tnay prove to be a valuable first step. 
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